13:59:11 <mackonstan> #startmeeting FD.io CSIT project meeting
13:59:11 <collab-meetbot`> Meeting started Wed Jul 14 13:59:11 2021 UTC and is due to finish in 60 minutes.  The chair is mackonstan. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:59:11 <collab-meetbot`> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:59:11 <collab-meetbot`> The meeting name has been set to 'fd_io_csit_project_meeting'
14:00:13 <tifrank> #info Tibor Frank
14:00:23 <pmikus> #info Peter Mikus
14:00:33 <vrpolak> #info Vratko Polak.
14:00:37 <fan_zhang> #info Fan Zhang
14:00:38 <mackonstan> #info mackonstan
14:00:41 <mackonstan> #chair
14:00:41 <collab-meetbot`> Current chairs: mackonstan
14:00:56 <mackonstan> #chair vrpolak
14:00:56 <collab-meetbot`> Warning: Nick not in channel: vrpolak
14:00:56 <collab-meetbot`> Current chairs: mackonstan vrpolak
14:01:26 <mackonstan> #topic Agenda bashing
14:02:12 <mackonstan> #topic Inputs from projects - VPP
14:03:09 <mackonstan> #info Andrew: vpp v21.01.1 maintenance is almost out
14:04:02 <mackonstan> #info Maciek: Need to agree what and when should be CSIT tested in v21.01.1.
14:04:25 <mackonstan> #topic Inputs from projects - VSAP
14:04:54 <mackonstan> #info No updates.
14:05:05 <mackonstan> #topic Inputs from projects - TSC
14:06:07 <mackonstan> #info Vratko: re TSC update, there will be election for committer-at-large. Ed is creating a mailer for FD.io committers only to facilitate the process.
14:06:44 <mackonstan> #info This will be followed by TSC appointing the members.
14:07:16 <mackonstan> #topic New performance tests - GTPU tests
14:07:31 <mackonstan> #topic New functional Tests
14:08:22 <dwallacelf> #info Dave Wallace
14:08:33 <mackonstan> #info Fan: re flow "offload" patch - would like to know the status of adding E810 NICs to vpp_device 1n-clx testbeds.
14:08:52 <mackonstan> #info Peter: still working on it.
14:10:21 <jlinkes> #info Juraj Linkeš
14:12:31 <mackonstan> #info Maciek: aside, will open the ticket tracking all new HW shipment status (one of the QSFP28 cables to be used for E810 in 1n-clx testbeds).
14:12:46 <mackonstan> #Fan: re performance tests, is still in progress.
14:14:30 <mackonstan> #info Juraj: re IPsec SPD entry tests, separate patches for inbound and outbound, 1 patch merged, 2 outstanding.
14:15:21 <mackonstan> #Vratko: correction 1 patch is in verify, not merged yet; the other two are in review.
14:15:49 <mackonstan> #topic Releases CSIT-2106
14:17:11 <mackonstan> #info Tibor: things looking quite well, almost all tests are finished. Only coverage 3n-skx, 2n-zn2, 3n-tsh are finishing, less than 50% remaining.
14:18:30 <mackonstan> #info Tibor: report status, some methodologies sections are missing, added TODO placeholders for methodology sections.
14:20:23 <mackonstan> #info Fan: content for gtpu test methodology and ipsec async crypto methodology is available, will push patches
14:22:02 <mackonstan> #info Maciek: here are the pointers to the right places in the report .rst files
14:22:16 <mackonstan> #info Maciek: gtpusw pointer
14:22:21 <mackonstan> #link https://git.fd.io/csit/tree/docs/report/introduction/methodology_tunnel_encapsulations.rst#n40
14:22:36 <mackonstan> #info Maciek: ipsec async pointer
14:22:58 <mackonstan> #link https://git.fd.io/csit/tree/docs/report/introduction/methodology_ipsec.rst#n50
14:25:52 <mackonstan> #info Tibor: everything else got merged, CSIT-2106 report generation job started 2hrs ago in rls2106 branch. Should be published within the next 2hrs.
14:28:31 <mackonstan> #into Tibor: maintenance report(s) will include following adds: 1) complete coverage tests as noted earlier, 2) af_xdp tests after they get added to job_specs, 3) TODO methodology sections, 4) experimental CSIT tests in AWS c5n environment.
14:31:38 <mackonstan> #info Vratko: two main regression RCAs under investigation: a) ipsec, b) nat44ed cps low scale (PDR results unstable). RCA markup would need to be added in the next maintenance version of csit-2106 report.
14:34:28 <mackonstan> #info Vratko: should mention performance issues with IPsec crypto async tests.
14:34:32 <fan_zhang> https://gerrit.fd.io/r/c/vpp/+/30386
14:36:28 <mackonstan> #info Fan: IPsec crypto async performance tests failing is related to this patch not being merged
14:36:36 <mackonstan> #link https://gerrit.fd.io/r/c/vpp/+/30386
14:39:15 <mackonstan> #info Maciek: looks like IPsec async crypto VPP code is broken. Need to add this to CSIT-2106 rls notes.
14:40:35 <mackonstan> #info Vratko: IPsec async crypto VPP, some tests pass but with low performance, likely to be related to the same issue.
14:40:55 <mackonstan> #info Vratko: will test the aforementioned patch, to see if issue is gone.
14:45:05 <mackonstan> #info Discussion about how to catch important feature-enabling and bug-fix VPP patches that are neglected, and making the feature broken.
14:46:07 <mackonstan> #topic Release - CSIT-2101.1
14:46:26 <mackonstan> #info VPP v21.01.1 is almost out
14:46:42 <mackonstan> #info Need to decide about CSIT coverage ...
14:47:40 <mackonstan> #info Dave: last time we picked up a subset of tests, and run lower number of iterations for them.
14:48:11 <mackonstan> #info Dave: it was more a smoke-test than anything else.
14:50:25 <mackonstan> #info Vratko: proposal - use existing iterative job_specs, just run a single iteration, and see if results fall within the avg+/-stdev of the previous full report run. Would need to add a new type of comparison table.
14:50:44 <mackonstan> #info Maciek, Vratko: no coverage tests.
14:51:51 <mackonstan> #info Dave: if results fall outside the avg+/-stdev, then do RCAs.
14:53:02 <mackonstan> #info Tibor, Maciek: will take a closer look at scheduling CSIT-2101.1 tests.
14:54:20 <mackonstan> #info Tibor: MRR trending tests run at reduced cadence of twice a week at the moment. Need to analyse when we are returning to daily cadence.
14:56:31 <mackonstan> #topic Physical Infrastructure
14:57:32 <mackonstan> #info Maciek, Peter: hsw servers are to be fully decommissioned.
14:57:46 <mackonstan> #topic Virtual Infrastructure, CI/CD
14:58:32 <mackonstan> #info Dave, virtual infra and CI infra are stable. Vanessa is testing the Jenkins upgrade in the sandbox, assuming it's stable, the production maintenance is scheduled for 19-Jul.
14:59:19 <mackonstan> #Dave, checked with Vanessa, Vexxhost not ready to move Internet ingress nodes to YUL-1 until middle of August.
14:59:57 <mackonstan> #topic Trending
15:00:55 <vrpolak> #link https://gerrit.fd.io/r/c/vpp/+/32878/7#message-51b619758963db4106d55d3da07300ddbe6928b6  <-- scale tests fail in master
15:00:58 <mackonstan> #info Vratko, CSIT ip4 and ip6 scale test are failing as VAT functionality that is used has been removed from VPP code as part of refactor.
15:01:57 <mackonstan> #endmeeting