openSUSE:Heroes/Meetings/20180303 Summary

What

openSUSE Heroes offsite 2018 team meeting minutes

Where

SUSE Event Area, SUSE HQ, Nuremberg, Germany, Earth, Solar System, Milky Way, Universe

When

Friday, 2018-03-02 until Sunday, 2018-03-04

Who

• All-time

• Bernhard Wiedemann
• Christian Boltz
• Christian Müller
• Per Jessen
• Sarah Julia Kriesch
• Theo Chatzimichos
• Thorsten Bro

• Guests

• Richard Brown
• Ludwig Nussel

Topics

• Coordinator
• openVPN / Bridging network - Provo / Nuremberg
• Board topics
• SLE 11 / Leap 15 - migration of old SLE 11 hosts / migration to Leap 15 of Leap 42.3 hosts
• Mirrors
• Salt / Automation
• monitoring.o.o / status.o.o
• Sponsoring
• Release Management
• Lessons learned

Agenda

Coordinator

• We need to organize ourselves a bit different, with the loss of Lars.
• In general we don’t want to have the role of “the Coordinator”, as it is not really clear what the responsibilites should/would be
• We need people who take-over responsibility for several tasks, but we want to organize ourselves managed by our ticket system
• For special purposes - we can define single responsibilities for tasks, e.g.:

• Organize off-site or other events [choose per event in Heroes Meeting]
• Talk / coordinate with Release Management and Board [Per Jessen]
• Hardware / Budget / Sponsoring [Thorsten Bro]
• Interface-Function to SUSE-IT / MF-IT and SUSE-people in general [Theo Chatzimichos]
• Heroes security (Sticking to coorporate SUSE / MF guidelines for security certification) [Christian Müller]
• Onboarding / Guidance of new openSUSE Heroes (Help needed! Ask Theo!) [Theo Chatzimichos, Christian Müller]

• We need an onboarding wiki page [Theo Chatzimichos]

• In the Heroes meeting - Check the oldest tickets every month - to not loose track [everybody]

openVPN / Bridging network - Provo / Nuremberg

• PRV <—> NUE - ToDo:

• Setup transfer network [rwawrig, cmueller]
• Setup VPN in Provo [tampakrap]
• HA-setup of scar + scarface [mcaj]
• Add separate gateway machines in Provo/Nuremberg to do routing (not on scar which is facing external) [tampakrap, mcaj]
• FreeIPA Master to Master setup (Provo / Nuremberg) [?]
• Master-to-Master setup DB? [?]

Board topics

• Boards wants us to be more public - actually putting our SALT / Documentation on public pages

• The onboarding process should be a starting point, to see which things we can document public and publish to the outside world
• SALT states (w/o pillars) will go public
• Documentation is up to everybody, to make it more public on its own extent

• connect.o.o should be replaced in future - this is a long-term ToDo for the next year

• check possibility of moving those extra DB-fields into eDirectory directly

• Start communication with Heinlein regarding pushing out mailservice, based on results of connect/eDirectory project

SLE 11 / Leap 15 - migration of old SLE 11 hosts / migration to Leap 15 of Leap 42.3 hosts

SLE 11 hosts upgrade to 42.3

• (XX) progress.o.o [tampakrap]
• (X) connect.o.o (boosters) - WAIT for more info

• create disconnect.o.o to disconnect the openSUSE TSP web service from connect.o.o machine / dependencies

• (~)old mysql

• Wiki-DB [cboltz, tbro] - Move Wiki-DB, Add MySQL-User-Handling to Salt

• (EASY)old postgresql [tampakrap]
• (EASY)narwal - servers (static.o.o, studio express, more) [tbro] -> Move to Leap15 beta
• (X)icc.o.o [tbro] (ask Kai about migration)
• (X)conference.o.o [Henne and OSEM maintainers]
• (X)community (irc-bot, some websites, maybe more stuff, etc.) [tampakrap]

• tickets are there to be done: find out if stuff can be moved to static.o.o or needs a new machine
• IRC-BOT (bugbot) - tampakrap will ask Henne about this

• (EASY) osc-collab [mcaj]

Leap 15

• Testing Leap 15 BETA

• Kiwi - Images for Leap 15 [tbro, cboltz]
• Salt - Leap 15 Repos [cboltz, bmwiedemann]
• Below tasks are blocked by the above:

• static.o.o (narwal) move to Leap 15 beta [tbro, see above]
• osc-collab to Leap 15 beta [mcaj, see above]

• community static pages and bugbot to Leap 15 [tampakrap]

• After the Leap 15 GM Release

• salt-master [tampakrap]
• All other web services [all heroes]
• Move mlmmj to mailman with testing first ;) [pjessen]

Mirrors

• pontifex2.i.o.o in NUE
• pontifex.i.o.o (rename the machine) [tampakrap]
• shutdown pontifex3 and be on alert if something breaks (old SLE11) [tampakrap]
• fix current mirroring setup

• fix reverse-DNS for all pontifex machines [mcaj, pjessen, tampakrap]
• do documentation of whole mirroring setup [mcaj, pjessen, tampakrap]
• fix monitoring of mirroring [mcaj, pjessen, tampakrap]

• work on mirrors as onboarding setup

• add mirroring servers to salt

• Widehat tasks

• widehat maybe out of date
• replace widehat with new sponsor [tbro]
• put widehat to infra.o.o VPN (client) [tampakrap]

• ask Ludwig about his publishing scripts
• reply fast with a “stock message” to mirroring requests (generic for other simple customer requests)

General advice

• Subscribe to admin-auto@o.o mailing list with your account
• make you machines / services (root-accounts) sending mails to the above mentioned list

Salt / Automation

• More configs / services

• Formulas preferred
• Static files / templates acceptable

• salt-master vs. monitoring machine lists are deviating

• monitoring client configs should go to the “base” role so that all machines get basic monitoring
• We need the packages / configs / services / NRPE / check_mk / etc.
• Check the “deploy CM” doc and see what can be automated

• Webpage git repo -> push -> CI -> Reactor -> git pull on minion

• GitHub repos cannot access gitlab-ci [bmwiedemann ask jdsn about mirroring git]

• Encrypted GPG Pillars

• documentation [tchatzichmichos]
• Missing script to re-encrypt all GPG pillars

• new services should always be configured with Salt!
• master in Provo (syndic)

monitoring.o.o / status.o.o

status.o.o [tampakrap, tbro]

• Users and admin access
• Documenation / ReadMe needs to be done
• Add new users to status.o.o while Onboarding
• Script to check if ACL in tickets, status, FreeIPA agree
• Replace with static page

• No DB, only Git
• No syncing
• DNS round-robin will be possible
• Git syncing:

• GitLab as central
• Minimal gitolite mirror on each status instance
• Multiple remotes on admin checkouts

• Problem: What about mail checks
• Check how Gentoo, Fedora and others handle it

• status page by Fedora: https://status.fedoraproject.org/ (maually updated) -> https://fedoraproject.org/wiki/Infrastructure#Infrastructure_health_status
• status page by Gentoo: https://infra-status-na1.gentoo.org/ -> https://wiki.gentoo.org/wiki/Project:Infrastructure/Servers (server overview)

• Current system

• Update both: Incident and Service
• Check documentation and update it [mcaj, AdaLovelace]

monitoring.o.o [mcaj, AdaLovelace]

• Automatic update of status.o.o through monitoring

• Monitoring to send events to status
• Status to check periodically if monitoring is accessible
• Send events maybe through Salt exec modules

• Send messages via Salt to Icinga with the status.$function exec module
• collect status with nrpe / checkmk / salt -> send json to icinga
• Add server to salt and add its configs / services
• All clients need also basic setup via Salt
• Monitor from external (Master and Satelite) / (per offered a machine)
• NTP to chrony (check_mk offers it)
• icinga1 to icinga2 migration
• Monitoring in Provo
• Thruk to combine Nue/Prv/External

Sponsoring [tbro]

• HW-Wishlist / Colocation-Space / Rootservers -> Write a wishlist [tbro]
• cpanel sponsored us this year with new 10Gig converged network adapters - thanks a lot!
• general sponsorship contact is Doug - who always should be the first contact

• Silver / Gold / Platinum states can be found on: https://en.opensuse.org/Sponsors

• We found out, that maybe not all sponsors are always up-to-date (search.o.o, build.o.o, opensuse.o, etc.)

• We want to provide a service: sponsors.o.o where you can get

• all sponsor logos via a URL in standard format / designs which fit in opensuse-design
• deliver a random sponsor logo: http://nginx.org/en/docs/http/ngx_http_random_index_module.html
• talk to Doug, designers and setup such a service [tbro]

• We took part in the TK-Award from ThomasKrenn

• Unfortunately, we were not in the winners’ list :’(

• Find a replacement with root-servers, rackspace or CDN-solution for widehat.o.o (talk to BuildOPS) [tbro]
• Evaluate CDN77 possibilities [cmueller]
• Heinlein -> proceed with mailsetup of opensuse.org and evaluate possibilities [cmueller]

Release Management

• progress.o.o - update without plugins
• pontifex monitor bandwith - would be helpful to find bugs
• openSUSE / publishing-scripts (on GitHub)

• they are used for Leap15 only - at the moment
• Tumbleweed should switch to them as well in future

• Piwik should be split or moved - ask all relevant teams
• Release (Leap15) will happen after oSC18

Lessons learned

• This meeting was more focused on maintenance and filling up the ToDo list for the future, in our first meeting in 2016, we were more focused on learning about infrastructure
• Well organized, no presentation missing, here are things to be done - and here are the people who do it
• Presentations about major topics: Network, Cloud, Mirror infrastructure, Salt were not given, but were not really missing
• We want at least one more in-person meeting during the year - one per year is not enough

• Plans are after the oSC (summer and beyond)

• On the next meeting, we should have reports of the splitted responsibilities, everybody who took over a responsibility should give a 10-15min overview as a status report about the “specialists topic” what the responsible was working on in the last months
• We should maybe increase the communication on #openSUSE-admin @ Freenode - it is not really busy and we should maybe talk about more topics there

Future topics

• Do we want to have the “traditional presentation” at the oSC18 like every year?

• We don’t want to have the “traditional presentation” at oSC18
• Theo wants to give a presentation on Salt testing in the openSUSE infrastructure
• Overview of the Heroes could be inside the last board talk
• We want to do a short advertisement at the oSC18 Lightning talks - it will be 5-10 minutes talk - saying who we are and that we’re hiring