13:00:27 <jki> #startmeeting CIP IRC weekly meeting
13:00:27 <brlogger`> Meeting started Thu Nov 25 13:00:27 2021 UTC and is due to finish in 60 minutes.  The chair is jki. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:00:27 <brlogger`> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:00:27 <brlogger`> The meeting name has been set to 'cip_irc_weekly_meeting'
13:00:31 <jki> hi all!
13:00:44 <patersonc[m]> Afternoon
13:01:02 <masami> Hi!
13:01:36 <alicef> hi
13:02:02 <iwamatsu> hi
13:03:21 <uli> hi
13:04:27 <jki> #topic AI review
13:04:33 <jki> 1. Combine root filesystem with kselftest binary - iwamatsu & alicef
13:05:44 <alicef> kselftest as been enabled on KernelCI for CIP and the pull request merged
13:06:08 <alicef> so we should see kselftest checks from now on
13:06:37 <alicef> main target is 5.10 as under 5.10 are not currently supported by kernelci
13:07:12 <alicef> the staging kselftest result can be seen here: https://staging.kernelci.org/test/job/kernelci/branch/staging-cip/kernel/staging-cip-20211124.1/
13:08:23 <alicef> I'm currently testing preempt_rt checks on the rt branches and sended a pull request for enabling it
13:08:35 <jki> how to the results compare to what should be expected?
13:08:57 <alicef> about kselftest ?
13:09:08 <alicef> what you should expect ?
13:09:11 <jki> yep, on that page you shared - known failures?
13:10:01 <alicef> if you see in the available test plans, you should see also kselftest_* entry
13:11:07 <alicef> and if you click on each one you can see the board/configuration that are working or giving warning
13:11:44 <jki> so, orange generally mean that a board is not available?
13:12:46 <alicef> no means that some test got a warning
13:15:30 <alicef> or failed https://staging.kernelci.org/test/plan/id/619ebd50dc20bbf117176107/
13:16:04 <alicef> it means that partial test set couldn't pass
13:16:21 <jki> ok, so the expected result would be all green, and those others need to be looked at?
13:16:32 <alicef> yes
13:16:57 <jki> and how do we compare here to the corresponding LTS version?
13:17:15 <alicef> on the kernelci mainling list we are also getting the cip report with regression or improvements
13:17:43 <alicef> is not comparing with LTS version. is comparing with previous cip test
13:18:04 <alicef> currently is just started to test kselftest on cip kernel
13:18:22 <alicef> so we don't have a base for understand if we had regression or not
13:19:38 <jki> ok, understood, thanks
13:20:03 <alicef> but by testing more we will some more information of what is happening
13:20:18 <alicef> will have
13:21:17 <alicef> I don't remeber if currently cip-dev is getting any kernelci mail
13:21:55 <jki> https://lists.cip-project.org/g/cip-dev/message/6812 - this one e.g.?
13:22:32 <jki> but none anymore since then
13:22:33 <alicef> yes
13:23:11 <alicef> right. I think it need to be fixed
13:23:18 <pave1> alicef: yes, cip-dev was getting it at some point.
13:23:29 <pave1> alicef: but I don't think we need to get those mails.
13:23:47 <pave1> alicef: Without detailed understanding of kernelci they are just noise.
13:23:50 <alicef> why not ? are not useful information ?
13:24:39 <pave1> alicef: I tried to understand them at one point and could not figure out what is wrong.
13:24:43 <alicef> that are the same mail format that also linux kernel is getting and using
13:25:39 <pave1> alicef: It is not just the mail format. Lets take https://lists.cip-project.org/g/cip-dev/message/6812 as an example.
13:26:22 <pave1> alicef: Sleep failure on some HP notebook we don't use. Is it just a fluke or real failure?
13:26:32 <pave1> alicef: And if it is real, how do we debug it?
13:27:22 <pave1> alicef: If you could select some real-looking failure that looks worth fixing, maybe we can work together to fix it? That would teach me how to work with that...
13:27:52 <alicef> that is still one of the many problem that it could catch
13:28:11 <alicef> dosen't mean that all problems are useful or fixable but some are
13:28:16 <iwamatsu> do we need check results on kernelci? I think it is necessary to check the test result on the cip side before kernelci.
13:28:25 <alicef> and you can get some good information about regression
13:29:08 <pave1> alicef: Yes maybe there are some good reports. But at least I'm lost in the kernelci reports and would need help if we tried to address some.
13:30:33 <alicef> also KernelCI is testing on the same board of cip side
13:30:48 <alicef> as KernelCI is using lab cip
13:31:13 <alicef> and that makes cip side work redundant
13:31:49 <iwamatsu> But kernelci does not use seme kernel config and same rootfs, right?
13:31:55 <alicef> as is not filtered only to lab cip. you can get also board and notebook results that are not on lab cip
13:32:13 <alicef> iwamatsu: it does better than cip side
13:32:28 <patersonc[m]> Is it not easier to work out regressions from the automated KernelCI emails then it is from looking through every single LAVA job as you have to now in the CIP setup?
13:32:33 <alicef> cip side rootfs is not updated instead kernelci side is using isar-cip-core
13:32:52 <alicef> and also updating it
13:33:27 <alicef> and 4.19 kernel config are getted correctly from the cip gitlab configuration repository
13:33:29 <iwamatsu> alicef: right. we need to update rootfs for testing.
13:33:42 <pave1> patersonc: I'm just looking for green marks in gitlab :-). Maybe kernelci would make the job easier, but I just don't know how to use it.
13:33:47 <alicef> kernelci is already doing it
13:34:16 <alicef> cip side core image are really old and not much relevant anymore. if you want to move to isar-cip-core
13:35:11 <alicef> also cip side is "afaik" not using configuration from the cip configuration repository
13:35:20 <jki> are really all boards we have in our lab also available to kernelci?
13:35:25 <alicef> but using the configuration from inside the kernel tree
13:35:45 <alicef> almost all. some are in progress of activation
13:35:52 <iwamatsu> yes, I thought it probably needed to work with the cip-core team. Until now, it was prepared by the kernel team...
13:35:53 <alicef> sended a pr for that
13:36:00 <jki> ok, good
13:36:19 <patersonc[m]> iwamatsu: This is what Alice is working on
13:36:34 <alicef> I setted kernelci mostly following as your request
13:36:51 <jki> so, then then major issue is understanding kernelci, like your results were understood?
13:37:03 <iwamatsu> patersonc: I see.
13:38:18 <alicef> mostly is reading them and checking what is possible to fix and discard what you are not interested
13:39:08 <patersonc[m]> pave1: The green marks just say that a LAVA job got to the end. It won't tell you if there are regressions in the actual tests.
13:39:33 <patersonc[m]> We'll have to do a proper walk-through of the KernelCI GUI
13:39:50 <patersonc[m]> The KernelCI project would appreciate any feedback to improve it
13:40:23 <alicef> patersonc[m]: I think reading mail should be the first priority, as mail are more explicit about what is failing and regression
13:41:30 <pave1> alicef: Could we get someone from testing team read the KernelCI reports and tell us if we have something to fix for now?
13:41:39 <jki> so, we only tested CIP boards, kernelci is now testing other boards as well - could step #1 be filtering out errors from those others for now?
13:41:46 <iwamatsu> Would we like to add a test result comparison function to the result display of gitlab?
13:43:00 <alicef> pave1: I think just keep a eye on what is failing and what is possible to fix is enough
13:43:12 <alicef> I don't know what you can and cannot fix
13:44:01 <pave1> alicef: If we have regression between 4.19.X and 4.19.X-cip, we definitely want to fix that.
13:44:36 <alicef> so after sending commits to 4.19 you should check the mail from cip about 4.19
13:44:49 <pave1> alicef: If we have a regression between 4.19.X and 4.19.X+1, we probably care, too.
13:44:53 <iwamatsu> pave1: agree.
13:44:54 <alicef> it will write found regression something about this
13:45:03 <patersonc[m]> Sure
13:47:11 <jki> so, what would be the next step?
13:47:15 <iwamatsu> I can prepare a comparison between 4.19.y and 4.19.y-cip.
13:47:50 <alicef> it will write something like 144 runs, 1 regressions
13:48:57 <alicef> a found
13:49:00 <alicef> [cip-dev] cip/linux-4.19.y-cip sleep: 8 runs, 2 regressions (v4.19.209-cip59) #kernelc
13:49:07 <jki> iwamatsu: thanks, recorded
13:49:35 <alicef> [cip-dev] cip/linux-4.19.y-cip baseline-nfs: 19 runs, 3 regressions (v4.19.209-cip59) #kernelci
13:50:08 <alicef> and from now on you should get more as we are starting to implement more test cases
13:50:22 <pave1> alicef: One problem is that it seems to be generating regressions between v4.19.200-cip10 and 4.19.210-cip11, while we'd need report between 4.19.210 and 4.19.210-cip11.
13:51:15 <jki> i think we need lts->lts+1 AND cip->cip+1
13:51:26 <alicef> regression from completely different tree ?
13:52:06 <iwamatsu> We can do it if we know LAVA job id for each kernel.
13:52:07 <jki> and then we can do the cross-comparison lts-cip
13:52:14 <alicef> maybe you can check lts report for 4.19.210 and compare it with 4.19.210-cip11 ?
13:52:47 <alicef> yes probably that should work
13:56:11 <jki> ok, so are settled? iwamatsu will do that initial comparison, and then we discuss how to continue?
13:56:27 <iwamatsu> jki: ok
13:57:15 <alicef> ok. iwamatsu san feel free to ping me if you need help with the comparison.
13:57:34 <iwamatsu> alicef: thanks.
13:57:46 <jki> perfect - thank you all!
13:57:57 <jki> continue? :)
13:58:48 <jki> alicef: just don't forget to refresh your PR for isar-cip-core so that we can close that
13:59:07 <jki> 2. Look into S3 artifact upload issues - patersonc
14:00:30 <patersonc[m]> WIP
14:00:45 <jki> good
14:00:48 <jki> #topic Kernel maintenance updates
14:01:42 <pave1> I did reviews: 5.10.80, 81, 82.
14:01:49 <uli> reviewing 5.10.80
14:02:02 <masami> This week there is 2 new CVEs. I backported patch for CVE-2021-4001 to stable/5.10.
14:02:09 <masami> iwamatsu: pavel: thank you for your review.
14:03:29 <iwamatsu> I reviewed 5.10.81 and 5.10.82-rc
14:04:34 <jki> is the new workflow with its wiki pages already established?
14:05:25 <iwamatsu> not yet. I will prepare and report in ML.
14:05:40 <jki> ok, thanks!
14:05:49 <jki> anything else for this topic?
14:06:04 <jki> 3
14:06:06 <jki> 2
14:06:09 <jki> 1
14:06:12 <jki> #topic Kernel testing
14:06:20 <jki> we can likely skip this :)
14:06:27 <patersonc[m]> yep :P
14:06:29 <alicef> we already discussed
14:06:34 <jki> #topic AOB
14:07:07 <jki> anyone anything?
14:07:26 <jki> 3
14:07:27 <patersonc[m]> Extended TSC tomorrow :)
14:07:32 <alicef> 13,99patersonc[m
14:07:33 <jki> jep!
14:07:46 <alicef> what is the situation with pr about 13,99patersonc[m
14:08:03 <alicef> https://github.com/orgs/kernelci/projects/11
14:08:22 <alicef> ?
14:09:06 <patersonc[m]> alicef: PR?
14:09:07 <alicef> about PR for the kernelci panel for organizing CIP
14:09:19 <alicef> like tweet or similar
14:09:53 <jki> PR like public relations, not pull request :)
14:10:03 <alicef> yes ! sorry
14:10:10 <jki> Neal would take input on that and tweet that
14:10:20 <alicef> nice thanks
14:10:43 <jki> anything else?
14:10:50 <alicef> i think would be nice to tweet about kernelci/cip effort
14:11:39 <jki> yes, please make a proposal, alicef!
14:12:00 <patersonc[m]> yep
14:12:51 <jki> great
14:13:07 <jki> then let's close?
14:13:13 <jki> 3
14:13:16 <jki> 2
14:13:19 <jki> 1
14:13:20 <jki> #endmeeting