openSUSE:Heroes/Meetings/20180303 Summary

Jump to: navigation, search


openSUSE Heroes offsite 2018 team meeting minutes


SUSE Event Area, SUSE HQ, Nuremberg, Germany, Earth, Solar System, Milky Way, Universe


Friday, 2018-03-02 until Sunday, 2018-03-04


  • All-time
  • Bernhard Wiedemann
  • Christian Boltz
  • Christian MĂźller
  • Per Jessen
  • Sarah Julia Kriesch
  • Theo Chatzimichos
  • Thorsten Bro
  • Guests
  • Richard Brown
  • Ludwig Nussel


  • Coordinator
  • openVPN / Bridging network - Provo / Nuremberg
  • Board topics
  • SLE 11 / Leap 15 - migration of old SLE 11 hosts / migration to Leap 15 of Leap 42.3 hosts
  • Mirrors
  • Salt / Automation
  • monitoring.o.o / status.o.o
  • Sponsoring
  • Release Management
  • Lessons learned


Introduction round
  • We need to organize ourselves a bit different, with the loss of Lars.
  • In general we don’t want to have the role of “the Coordinator”, as it is not really clear what the responsibilites should/would be
  • We need people who take-over responsibility for several tasks, but we want to organize ourselves managed by our ticket system
  • For special purposes - we can define single responsibilities for tasks, e.g.:
  • Organize off-site or other events [choose per event in Heroes Meeting]
  • Talk / coordinate with Release Management and Board [Per Jessen]
  • Hardware / Budget / Sponsoring [Thorsten Bro]
  • Interface-Function to SUSE-IT / MF-IT and SUSE-people in general [Theo Chatzimichos]
  • Heroes security (Sticking to coorporate SUSE / MF guidelines for security certification) [Christian MĂźller]
  • Onboarding / Guidance of new openSUSE Heroes (Help needed! Ask Theo!) [Theo Chatzimichos, Christian MĂźller]
  • We need an onboarding wiki page [Theo Chatzimichos]
  • In the Heroes meeting - Check the oldest tickets every month - to not loose track [everybody]
openVPN / Bridging network - Provo / Nuremberg
  • PRV <—> NUE - ToDo:
  • Setup transfer network [rwawrig, cmueller]
  • Setup VPN in Provo [tampakrap]
  • HA-setup of scar + scarface [mcaj]
  • Add separate gateway machines in Provo/Nuremberg to do routing (not on scar which is facing external) [tampakrap, mcaj]
  • FreeIPA Master to Master setup (Provo / Nuremberg) [?]
  • Master-to-Master setup DB? [?]
Board topics
  • Boards wants us to be more public - actually putting our SALT / Documentation on public pages
  • The onboarding process should be a starting point, to see which things we can document public and publish to the outside world
  • SALT states (w/o pillars) will go public
  • Documentation is up to everybody, to make it more public on its own extent
  • connect.o.o should be replaced in future - this is a long-term ToDo for the next year
  • check possibility of moving those extra DB-fields into eDirectory directly
  • Start communication with Heinlein regarding pushing out mailservice, based on results of connect/eDirectory project
SLE 11 / Leap 15 - migration of old SLE 11 hosts / migration to Leap 15 of Leap 42.3 hosts
SLE 11 hosts upgrade to 42.3
  • (XX) progress.o.o [tampakrap]
  • (X) connect.o.o (boosters) - WAIT for more info
  • create disconnect.o.o to disconnect the openSUSE TSP web service from connect.o.o machine / dependencies
  • (~)old mysql
  • Wiki-DB [cboltz, tbro] - Move Wiki-DB, Add MySQL-User-Handling to Salt
  • (EASY)old postgresql [tampakrap]
  • (EASY)narwal - servers (static.o.o, studio express, more) [tbro] -> Move to Leap15 beta
  • (X)icc.o.o [tbro] (ask Kai about migration)
  • (X)conference.o.o [Henne and OSEM maintainers]
  • (X)community (irc-bot, some websites, maybe more stuff, etc.) [tampakrap]
  • tickets are there to be done: find out if stuff can be moved to static.o.o or needs a new machine
  • IRC-BOT (bugbot) - tampakrap will ask Henne about this
  • (EASY) osc-collab [mcaj]
Leap 15
  • Testing Leap 15 BETA
  • Kiwi - Images for Leap 15 [tbro, cboltz]
  • Salt - Leap 15 Repos [cboltz, bmwiedemann]
  • Below tasks are blocked by the above:
  • static.o.o (narwal) move to Leap 15 beta [tbro, see above]
  • osc-collab to Leap 15 beta [mcaj, see above]
  • community static pages and bugbot to Leap 15 [tampakrap]
  • After the Leap 15 GM Release
  • salt-master [tampakrap]
  • All other web services [all heroes]
  • Move mlmmj to mailman with testing first ;) [pjessen]
  • pontifex2.i.o.o in NUE
  • pontifex.i.o.o (rename the machine) [tampakrap]
  • shutdown pontifex3 and be on alert if something breaks (old SLE11) [tampakrap]
  • fix current mirroring setup
  • fix reverse-DNS for all pontifex machines [mcaj, pjessen, tampakrap]
  • do documentation of whole mirroring setup [mcaj, pjessen, tampakrap]
  • fix monitoring of mirroring [mcaj, pjessen, tampakrap]
  • work on mirrors as onboarding setup
  • add mirroring servers to salt
  • Widehat tasks
  • widehat maybe out of date
  • replace widehat with new sponsor [tbro]
  • put widehat to infra.o.o VPN (client) [tampakrap]
  • ask Ludwig about his publishing scripts
  • reply fast with a “stock message” to mirroring requests (generic for other simple customer requests)
General advice
  • Subscribe to admin-auto@o.o mailing list with your account
  • make you machines / services (root-accounts) sending mails to the above mentioned list
Salt / Automation
  • More configs / services
  • Formulas preferred
  • Static files / templates acceptable
  • salt-master vs. monitoring machine lists are deviating
  • monitoring client configs should go to the “base” role so that all machines get basic monitoring
  • We need the packages / configs / services / NRPE / check_mk / etc.
  • Check the “deploy CM” doc and see what can be automated
  • Webpage git repo -> push -> CI -> Reactor -> git pull on minion
  • GitHub repos cannot access gitlab-ci [bmwiedemann ask jdsn about mirroring git]
  • Encrypted GPG Pillars
  • documentation [tchatzichmichos]
  • Missing script to re-encrypt all GPG pillars
  • new services should always be configured with Salt!
  • master in Provo (syndic)
monitoring.o.o / status.o.o
status.o.o [tampakrap, tbro]
  • Users and admin access
  • Documenation / ReadMe needs to be done
  • Add new users to status.o.o while Onboarding
  • Script to check if ACL in tickets, status, FreeIPA agree
  • Replace with static page
  • No DB, only Git
  • No syncing
  • DNS round-robin will be possible
  • Git syncing:
  • GitLab as central
  • Minimal gitolite mirror on each status instance
  • Multiple remotes on admin checkouts
  • Problem: What about mail checks
  • Check how Gentoo, Fedora and others handle it
  • Current system
  • Update both: Incident and Service
  • Check documentation and update it [mcaj, AdaLovelace]
monitoring.o.o [mcaj, AdaLovelace]
  • Automatic update of status.o.o through monitoring
  • Monitoring to send events to status
  • Status to check periodically if monitoring is accessible
  • Send events maybe through Salt exec modules
  • Send messages via Salt to Icinga with the status.$function exec module
  • collect status with nrpe / checkmk / salt -> send json to icinga
  • Add server to salt and add its configs / services
  • All clients need also basic setup via Salt
  • Monitor from external (Master and Satelite) / (per offered a machine)
  • NTP to chrony (check_mk offers it)
  • icinga1 to icinga2 migration
  • Monitoring in Provo
  • Thruk to combine Nue/Prv/External
Sponsoring [tbro]
  • HW-Wishlist / Colocation-Space / Rootservers -> Write a wishlist [tbro]
  • cpanel sponsored us this year with new 10Gig converged network adapters - thanks a lot!
  • general sponsorship contact is Doug - who always should be the first contact
  • We found out, that maybe not all sponsors are always up-to-date (search.o.o, build.o.o, opensuse.o, etc.)
  • We want to provide a service: sponsors.o.o where you can get
  • We took part in the TK-Award from ThomasKrenn
  • Unfortunately, we were not in the winners’ list :’(
  • Find a replacement with root-servers, rackspace or CDN-solution for widehat.o.o (talk to BuildOPS) [tbro]
  • Evaluate CDN77 possibilities [cmueller]
  • Heinlein -> proceed with mailsetup of and evaluate possibilities [cmueller]
Release Management
  • progress.o.o - update without plugins
  • pontifex monitor bandwith - would be helpful to find bugs
  • openSUSE / publishing-scripts (on GitHub)
  • they are used for Leap15 only - at the moment
  • Tumbleweed should switch to them as well in future
  • Piwik should be split or moved - ask all relevant teams
  • Release (Leap15) will happen after oSC18
Lessons learned
  • This meeting was more focused on maintenance and filling up the ToDo list for the future, in our first meeting in 2016, we were more focused on learning about infrastructure
  • Well organized, no presentation missing, here are things to be done - and here are the people who do it
  • Presentations about major topics: Network, Cloud, Mirror infrastructure, Salt were not given, but were not really missing
  • We want at least one more in-person meeting during the year - one per year is not enough
  • Plans are after the oSC (summer and beyond)
  • On the next meeting, we should have reports of the splitted responsibilities, everybody who took over a responsibility should give a 10-15min overview as a status report about the “specialists topic” what the responsible was working on in the last months
  • We should maybe increase the communication on #openSUSE-admin @ Freenode - it is not really busy and we should maybe talk about more topics there
Future topics
  • Do we want to have the “traditional presentation” at the oSC18 like every year?
  • We don’t want to have the “traditional presentation” at oSC18
  • Theo wants to give a presentation on Salt testing in the openSUSE infrastructure
  • Overview of the Heroes could be inside the last board talk
  • We want to do a short advertisement at the oSC18 Lightning talks - it will be 5-10 minutes talk - saying who we are and that we’re hiring