| *** shoragan <shoragan!~shoragan@user/shoragan> has quit IRC (*.net *.split) | 02:20 | |
| *** shoragan <shoragan!~shoragan@user/shoragan> has joined #cip | 02:22 | |
| *** rajm <rajm!~robert@cpc126990-macc4-2-0-cust43.1-3.cable.virginm.net> has joined #cip | 06:35 | |
| *** prabhakarlad <prabhakarlad!~prabhakar@pc.renesas.eu> has joined #cip | 07:38 | |
| *** prabhakarlad <prabhakarlad!~prabhakar@pc.renesas.eu> has quit IRC (Ping timeout: 244 seconds) | 08:27 | |
| *** prabhakarlad <prabhakarlad!~prabhakar@pc.renesas.eu> has joined #cip | 08:57 | |
| *** uli <uli!~uli@d5369b80.access.ecotel.net> has joined #cip | 10:50 | |
| *** masami <masami!~masami@FL1-125-198-44-131.tky.mesh.ad.jp> has joined #cip | 11:42 | |
| *** tobsch <tobsch!~quassel@195.145.170.188> has joined #cip | 11:46 | |
| *** pave1 <pave1!~pavel@jabberwock.ucw.cz> has joined #cip | 11:58 | |
| *** jki <jki!~jki@195.145.170.157> has joined #cip | 12:00 | |
| jki | hi there! | 12:00 |
|---|---|---|
| iwamatsu | hi | 12:00 |
| uli | hello | 12:00 |
| masami | hello | 12:00 |
| alicefm | Hi | 12:00 |
| patersonc[m] | Hello | 12:00 |
| pave1 | hi | 12:00 |
| jki | #startmeeting CIP IRC weekly meeting | 12:02 |
| collab-meetbot` | Meeting started Thu Oct 27 12:02:15 2022 UTC and is due to finish in 60 minutes. The chair is jki. Information about MeetBot at http://wiki.debian.org/MeetBot. | 12:02 |
| collab-meetbot` | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 12:02 |
| collab-meetbot` | The meeting name has been set to 'cip_irc_weekly_meeting' | 12:02 |
| *** collab-meetbot` changes topic to " (Meeting topic: CIP IRC weekly meeting)" | 12:02 | |
| jki | #topic AI review | 12:02 |
| *** collab-meetbot` changes topic to "AI review (Meeting topic: CIP IRC weekly meeting)" | 12:02 | |
| jki | 1. Add qemu-riscv to cip-kernel-config - patersonc | 12:02 |
| patersonc[m] | No updates | 12:02 |
| jki | too bad - we are close to zero AI ;) | 12:03 |
| jki | any other AIs? | 12:03 |
| patersonc[m] | Sorry! | 12:04 |
| jki | np | 12:04 |
| jki | then moving on... | 12:04 |
| jki | 3 | 12:04 |
| jki | 2 | 12:04 |
| jki | 1 | 12:04 |
| jki | #topic Kernel maintenance updates | 12:04 |
| *** collab-meetbot` changes topic to "Kernel maintenance updates (Meeting topic: CIP IRC weekly meeting)" | 12:04 | |
| uli | reviewing 5.10.150 | 12:05 |
| pave1 | Reviewing 5.10.150, 151. | 12:05 |
| masami | There is 20 CVEs reported. These are not so critical vulnerabilities. | 12:05 |
| iwamatsu | I reviewed 5.10.150. | 12:05 |
| jki | any other topic on maintenance? | 12:08 |
| jki | then: moving on... | 12:10 |
| jki | 3 | 12:10 |
| jki | 2 | 12:10 |
| jki | 1 | 12:10 |
| pave1 | Busy week with huge reviews from -rc1, sorry. | 12:10 |
| jki | ok :) | 12:10 |
| pave1 | We can move on. | 12:10 |
| jki | #topic Kernel testing | 12:10 |
| *** collab-meetbot` changes topic to "Kernel testing (Meeting topic: CIP IRC weekly meeting)" | 12:10 | |
| alicef_ | patersonc[m]: I wrote on #lava-docker about the lava security problem | 12:11 |
| patersonc[m] | Thanks, I saw it recently. We really need to upgrade our LAVA version anyway, this is a good kick | 12:11 |
| alicef_ | oh I just see your replay | 12:11 |
| iwamatsu | nice | 12:11 |
| patersonc[m] | I'll try and start investigating in the next few weeks, I just need to get a few internal things sorted first | 12:12 |
| alicef_ | thanks pave1 for sending the email with the warnings checks | 12:12 |
| patersonc[m] | +1 | 12:13 |
| pave1 | You are welcome :-). | 12:13 |
| pave1 | For the results, I believe good start would be reporting failure whenever test changes. | 12:14 |
| pave1 | I guess that will simply result on smc fails on qemu? We can disable those as a next step. | 12:14 |
| pave1 | Seeing one fail and investigating is better than having to go through all results every time to see if something maybe failed somewhere. | 12:15 |
| alicef_ | disable it only for qemu ? | 12:15 |
| pave1 | alicef: Yes. Or.. does it cause problems elesewhere? | 12:15 |
| alicef_ | I don't think so | 12:16 |
| pave1 | (Or it can be still run and result ignored, or something, if that's easier). | 12:17 |
| patersonc[m] | >For the results, I believe good start would be reporting failure whenever test changes. | 12:18 |
| patersonc[m] | This would mean setting up a database to keep track of previous test results etc. | 12:18 |
| patersonc[m] | Then comparing previous result to the new result, and only fail if it's changed from pass->fail | 12:18 |
| patersonc[m] | Is this what you mean? | 12:18 |
| patersonc[m] | * > For the results, I believe good start would be reporting failure whenever test changes. | 12:18 |
| patersonc[m] | This would mean setting up a database to keep track of previous test results etc. | 12:18 |
| patersonc[m] | Then comparing previous result to the new result, and only fail if it's changed from pass->fail | 12:18 |
| patersonc[m] | Is this what you mean? | 12:18 |
| pave1 | No, not really. | 12:18 |
| pave1 | "If the target is qemu && test is smc then ignore the test result". | 12:19 |
| patersonc[m] | Yes, but we get smc failures on other boards as well | 12:19 |
| pave1 | Ok, can I get an exampel? | 12:19 |
| patersonc[m] | https://gitlab.com/cip-project/cip-kernel/linux-cip/-/jobs/3141003858#L262 | 12:20 |
| pave1 | Can we run smc on corresponding upstream kernel, to verify we did not break anything, and then probably disable smc test? | 12:21 |
| pave1 | Thanks for example, I'll take a look. | 12:21 |
| patersonc[m] | Check the email I sent on Tuesday for more links | 12:22 |
| * patersonc[m] checks for more links now - 2 secs | 12:24 | |
| patersonc[m] | Upstream: https://linux.kernelci.org/test/plan/id/635a180f43c30ac29ce7db6f/ | 12:24 |
| patersonc[m] | CIP: https://gitlab.com/cip-project/cip-testing/linux-cip-ci/-/jobs/3224759608#L1132 | 12:25 |
| patersonc[m] | Both on the hihope-rzg2m board | 12:25 |
| patersonc[m] | Same CVE failure | 12:25 |
| patersonc[m] | (obviously different kernels) | 12:25 |
| pave1 | If the test fails even on 6.1-rc2, it will probably fail in 5.10-stable and 5.10-cip, too. | 12:26 |
| pave1 | Let's ignore the test. | 12:26 |
| pave1 | Going through the emails, you said: "I'd still need this job to return a green tick though if the actual job runs successfully, even if there are failed test cases. | 12:27 |
| iwamatsu | But xilinx MPSoC is no issue. https://lava.ciplatform.org/scheduler/job/771035#L859 | 12:27 |
| pave1 | Actually, I believe what we should do. | 12:27 |
| pave1 | This is not kernel issue, this is subtly broken hardware. | 12:27 |
| pave1 | Plus it is "security" and people love to say "this is critical" | 12:28 |
| pave1 | when it is not. | 12:28 |
| pave1 | We have security team. Could we simply ask our security team to investigate that? | 12:28 |
| pave1 | ...best in cooperation with hardware vendors. :-) | 12:29 |
| pave1 | Why do you want to put green ticks on failing tests? | 12:29 |
| pave1 | Lets simply use "green" -> nothing to see here. | 12:29 |
| pave1 | "red" -> someone needs to look at the results. | 12:29 |
| pave1 | I guess it is normally me looking at the results, and I escalate if I can't understand that, but don't make me examine logs for 12 targets when there's nothing to see there. | 12:30 |
| patersonc[m] | This was referring to if there are tests that always fail, such as those smc tests | 12:31 |
| pave1 | Ok, lets disable those on platforms where they always fail. | 12:32 |
| patersonc[m] | Is a view like this better? | 12:32 |
| patersonc[m] | https://linux.kernelci.org/test/job/cip/branch/linux-4.4.y-cip/kernel/v4.4.302-cip70-98-g7f7838c92740f/ | 12:32 |
| pave1 | Not really. When I tried to investigate a failure, I got into maze of results.. | 12:34 |
| pave1 | ...https://linux.kernelci.org/test/plan/id/62d01083f0ebb14854a39bde/ | 12:34 |
| pave1 | ...which seems to be caused by flakey target failing randomly. | 12:34 |
| patersonc[m] | But flakey targets is no different for kernelci compared to cip right? | 12:35 |
| pave1 | ...but because results are sorted by test (not target), it is not obvious to see. | 12:35 |
| patersonc[m] | okay | 12:35 |
| patersonc[m] | This view should show the targets that failed: https://linux.kernelci.org/test/job/cip-gitlab/branch/ci%2Fpavel%2Flinux-test/kernel/v5.10.147-cip18-11-ge6d27ea102c3b/plan/baseline/ | 12:36 |
| pave1 | It works as long as they are no flakey things. We are happy to have none in the gitlab. | 12:36 |
| patersonc[m] | s/failed/regressed/ | 12:36 |
| alicef_ | about the asus chromebook failures I asked on kernelci about that. and they was looking into such problems. not sure if they did something. | 12:36 |
| alicef_ | actually the best would be to open a issue on kernelci about boards that fails often, as it can be relevant also for other kernel tree | 12:37 |
| pave1 | So the way I use test branch (pushing 5.10, 4.19 and 4.4 into it) leads into really confusing stuff on kernelci. | 12:38 |
| patersonc[m] | Could you use a different branch for each kernel version? | 12:39 |
| patersonc[m] | Then the automated regression tracking would work | 12:40 |
| pave1 | I'll need to jump on a branch, too, during bisect for example. | 12:41 |
| pave1 | I'm afraid I'll get the regression tracking confused.. | 12:41 |
| jki | then you need a separate branch for debugging, vs. one per kernel that is moving forward only and tracks regressions | 12:43 |
| pave1 | Yes, that's current setup. | 12:44 |
| pave1 | Actually we have 4.4-st-rc and 4.4-st because more debugging is happening there. | 12:44 |
| patersonc[m] | Taking a step back a bit, why don't we change our GitLab CI approach to be more for development - "instant" CI, and mainly focus on kernel builds and boot tests. | 12:45 |
| patersonc[m] | Then use KernelCI for the full (pre)release testing with full build/test coverage. | 12:45 |
| patersonc[m] | GitLab CI can then provide simple green/red statuses, and KernelCI can track regressions properly etc. | 12:45 |
| patersonc[m] | Best of both worlds | 12:45 |
| patersonc[m] | * builds and (only) boot tests. | 12:46 |
| patersonc[m] | Then rather spend resources trying to make our gitlab CI setup do the same thing as kernelci, we can spend them on improving kernelci for all, and on improving things like platform stability in the labs | 12:46 |
| alicef_ | updating lava also | 12:47 |
| pave1 | Kernelci is quite confusing to me. Gitlab CI is a bit better, but hiding failures behind green ticks is not nice. | 12:48 |
| pave1 | We can split testing between kernelci and gitlab anyway you want, if you take over interpretting the results. | 12:49 |
| pave1 | I can simply ask over email, and testing team says "okay, that kernel looks good to us" or "hey there's bug you should debug". | 12:50 |
| patersonc[m] | Is anyone else in the kernel team looking at test results on KernelCI/gitlab? | 12:51 |
| uli | i don't, usually | 12:52 |
| iwamatsu | I am checking sometime. | 12:52 |
| jki | is the number of reports we get on cip-dev useful then, or can it be made useful? | 12:54 |
| pave1 | I don't find those reports useful. | 12:57 |
| patersonc[m] | From each build triggered on kernelci we get 1 email reporting on the build status, and 1 email per test suite. | 12:58 |
| patersonc[m] | The test emails have a section at the top which tells you which boards had regressions. There is a link to click in order to see the logs etc. | 12:58 |
| iwamatsu | Is there a lot of unnecessary information? | 12:58 |
| patersonc[m] | There's a lot more information if you scroll down, but it's direct links to each board that failed, rather then the summary link at the top | 13:00 |
| patersonc[m] | So you can either click the summary link and navigate the website, or click on the specific links. All useful depending on your choice | 13:01 |
| alicefm | Also already KernelCI works by regression and notify if a previous working test failed | 13:01 |
| pave1 | Well, there's buch of reports from kernelci on cip-dev from Oct 26. | 13:02 |
| alicefm | SMC problem got notified because on the previous 4.4-CIP builds it was not failings | 13:03 |
| pave1 | It reports lot of failures including regressions, but I doubt there's a real regression in 5.10-rc tree. | 13:03 |
| alicefm | Pavel that because KernelCI is sending a build report and a board/regression reports | 13:03 |
| pave1 | Could we get a red cross whenever gitlab detects something that can be debugged, please? | 13:04 |
| alicefm | Build report are always sent, board/regression reports are sent with problems | 13:05 |
| pave1 | I'm trying to get gitlab useful, first. kernelci on 5.10 will require more work. | 13:06 |
| patersonc[m] | Okay | 13:08 |
| patersonc[m] | Please take a look at my email from tuesday and comment on the MR etc. Then GitLab will be red if individual test cases fail | 13:08 |
| pave1 | Ok. | 13:08 |
| pave1 | Can we get smc disabled on problematic boards? | 13:09 |
| patersonc[m] | I'll have to check if any of the boards pass | 13:10 |
| patersonc[m] | May be easier to disable it alltogether if we're not interested in the results | 13:10 |
| patersonc[m] | But it's not just smc. We'll get failures in all the ltp tests as well | 13:10 |
| pave1 | Can we disable those ltp cases that always fail? | 13:11 |
| iwamatsu | These can be controlled with LAVA metadata. | 13:11 |
| pave1 | But I don't see ltp failures. | 13:12 |
| pave1 | I picked up random example: | 13:12 |
| pave1 | https://gitlab.com/cip-project/cip-testing/linux-stable-rc-ci/-/jobs/3235839704 | 13:12 |
| pave1 | and only see 0_spectre-meltdown-checker-test.CVE-2018-3640 [fail] there. | 13:12 |
| patersonc[m] | That test job only ran boot and smc tests | 13:12 |
| pave1 | When are we running ltp tests? | 13:13 |
| patersonc[m] | On rc testing | 13:14 |
| patersonc[m] | Here's an example: https://gitlab.com/cip-project/cip-kernel/linux-cip/-/jobs/3186281973 | 13:14 |
| pave1 | Fun. That will need some more work. | 13:15 |
| pave1 | But I'd still proceed with the red cross, so we know we have work to do. | 13:15 |
| patersonc[m] | Okay | 13:15 |
| pave1 | Thank you! | 13:16 |
| patersonc[m] | Anyway, the MR is there to comment on. | 13:16 |
| patersonc[m] | Let's move on because we're already well over time | 13:16 |
| jki | indeed :) | 13:16 |
| jki | 3 | 13:16 |
| jki | 2 | 13:16 |
| jki | 1 | 13:16 |
| jki | #topic AOB | 13:16 |
| *** collab-meetbot` changes topic to "AOB (Meeting topic: CIP IRC weekly meeting)" | 13:16 | |
| jki | anything else to discuss today? | 13:17 |
| jki | 3 | 13:18 |
| jki | 2 | 13:18 |
| jki | 1 | 13:18 |
| jki | #endmeeting | 13:18 |
| collab-meetbot` | Meeting ended Thu Oct 27 13:18:09 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:18 |
| collab-meetbot` | Minutes: http://ircbot.wl.linuxfoundation.org/meetings/cip/2022/10/cip.2022-10-27-12.02.html | 13:18 |
| collab-meetbot` | Minutes (text): http://ircbot.wl.linuxfoundation.org/meetings/cip/2022/10/cip.2022-10-27-12.02.txt | 13:18 |
| collab-meetbot` | Log: http://ircbot.wl.linuxfoundation.org/meetings/cip/2022/10/cip.2022-10-27-12.02.log.html | 13:18 |
| *** collab-meetbot` changes topic to "Civil Infrastructure Platform Project. CIP mailing list at https://lists.cip-project.org/g/cip-dev | CIP kernel meeting every Thursday at 12:00 UTC | Find the meeting logs at https://ircbot.wl.linuxfoundation.org/meetings/cip/ and chat logs at https://ircbot.wl.linuxfoundation.org/logs/%23cip/"" | 13:18 | |
| jki | then: thank you all! | 13:18 |
| uli | thanks | 13:18 |
| alicefm | Thank you | 13:18 |
| masami | thank you | 13:18 |
| iwamatsu | thank you | 13:18 |
| jki | patersonc[m]: any news from the rzfive and ldconfig? | 13:18 |
| pave1 | Thank you! | 13:19 |
| patersonc[m] | jki: Not yet, sorry. It is being looked at though | 13:23 |
| *** uli <uli!~uli@d5369b80.access.ecotel.net> has left #cip (Leaving) | 13:28 | |
| *** jki <jki!~jki@195.145.170.157> has quit IRC (Quit: Leaving) | 14:32 | |
| *** prabhakarlad <prabhakarlad!~prabhakar@pc.renesas.eu> has quit IRC (Quit: Client closed) | 15:09 | |
| *** tobsch <tobsch!~quassel@195.145.170.188> has quit IRC (Ping timeout: 244 seconds) | 15:14 | |
| *** tobsch <tobsch!~quassel@ip5f5bfaca.dynamic.kabel-deutschland.de> has joined #cip | 16:38 | |
| *** tobsch <tobsch!~quassel@ip5f5bfaca.dynamic.kabel-deutschland.de> has quit IRC (Ping timeout: 244 seconds) | 16:46 | |
| *** tobsch <tobsch!~quassel@2a02:810d:4bc0:2020:7b79:e9a7:3687:726f> has joined #cip | 16:46 | |
| *** tobsch <tobsch!~quassel@2a02:810d:4bc0:2020:7b79:e9a7:3687:726f> has quit IRC (Ping timeout: 255 seconds) | 17:02 | |
| *** tobsch <tobsch!~quassel@2a02:810d:4bc0:2020:e1b:11c:aca3:c3c8> has joined #cip | 17:13 | |
| *** tobsch <tobsch!~quassel@2a02:810d:4bc0:2020:e1b:11c:aca3:c3c8> has quit IRC (Ping timeout: 250 seconds) | 17:31 | |
| *** prabhakarlad <prabhakarlad!~prabhakar@pc.renesas.eu> has joined #cip | 19:05 | |
| iwamatsu | bry449dj | 20:39 |
| iwamatsu | omg | 20:41 |
| *** prabhakarlad <prabhakarlad!~prabhakar@pc.renesas.eu> has quit IRC (Quit: Client closed) | 21:05 | |
| *** prabhakarlad <prabhakarlad!~prabhakar@pc.renesas.eu> has joined #cip | 21:09 | |
| *** rajm <rajm!~robert@cpc126990-macc4-2-0-cust43.1-3.cable.virginm.net> has quit IRC (Ping timeout: 244 seconds) | 21:52 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!